Histomorphometric classifier to predict cardiac failure from whole-slide hematoxylin and eosin stained images

ABSTRACT

Methods, apparatus, and other embodiments predict heart failure from WSIs of cardiac histopathology using a deep learning convolutional neural network (CNN). One example apparatus includes a pre-processing circuit configured to generate a pre-processed WSI by downsampling a digital WSI; an image acquisition circuit configured to randomly select a set of non-overlapping ROIs from the pre-processed WSI, and configured to provide the set of non-overlapping ROIs to a deep learning circuit; a deep learning circuit configured to generate an image-level probability that a member of the set of non-overlapping ROIs is a failure/abnormal pathology ROI using a CNN; and a classification circuit configured to generate a patient-level probability that the patient from which the region of tissue represented in the WSI was acquired is experiencing failure or non-failure based, at least in part, on the image-level probability.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application62/417,458 filed Nov. 4, 2016.

FEDERAL FUNDING NOTICE

The invention was made with government support under National CancerInstitute of the National Institutes of Health award numbers(R21CA179327-01, R21CA195152-01, 1U24CA199374-01, R01CA202752-01A1,R01CA208236-01A1), the National Institute of Diabetes and Digestive andKidney Diseases under award number R01 DK098503-02, the National Centerfor Research Resources under award number 1 C06 RR12463-01, the NationalHeart Lung and Blood Institute under award number R01-HL105993, the DODProstate Cancer Synergistic Idea Development Award (PC120857); the DODLung Cancer Idea Development New Investigator Award (LC130463), the DODProstate Cancer Idea Development Award, the DOD Peer Reviewed CancerResearch Program W81XWH-16-1-0329, and the National Institute ofNeurological Disorders and Stroke F30NS092227. The government hascertain rights in the invention.

BACKGROUND

Cardiovascular diseases are the leading cause of death globally, and theleading cause of hospital admissions in the U.S. and Europe. More than26 million people worldwide suffer from heart failure each year, andabout half of these patients die within five years. Clinical heartfailure is a progressive syndrome where impaired ventricular functionresults in inadequate systemic perfusion. The diagnosis of heart failureconventionally relies on clinical history, physical examination, basiclab tests, and imaging. This diagnostic rubric has not changed in threedecades, and lacks the ability to accurately sub-classify patients intothe numerous potential clinical etiologies, which in turn has limitedthe development of new treatments. However, when the cause of heartfailure is unidentified, endomyocardial biopsy (EMB) represents the goldstandard for evaluation and grading of heart disease.

Conventional approaches to analyzing EMBs are not optimal. Manualinterpretation of EMB suffers from high inter-rater variability in thepathologic diagnosis of heart disease. Manual interpretation of EMBs byexpert human pathologists has an accuracy of only approximately 75% whenclassifying an EMB as indicating heart failure or non-heart failure.Furthermore, manual interpretation of EMB has limited clinicalindications. Meanwhile, the increasingly common digitization of glasspathology slides has lead to a proliferation of whole-slide imaging(WSI) platforms.

Conventional image analysis approaches that employ digitized WSI imagesmay involve manually engineering features. The manually engineeredfeatures may include intensity statistics, texture descriptors, or imagedecompositions. These features are then provided to a supervised machinelearning algorithm for classification or regression. Designingdiscriminative features is a long process that requires computationalexperience and domain knowledge to develop features that might,potentially, be relevant to the intended classification. Furthermore,designing discriminative features may leave out relevant or evencurrently biologically unknown features.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various example apparatus,methods, and other example embodiments of various aspects of theinvention. It will be appreciated that the illustrated elementboundaries (e.g., boxes, groups of boxes, or other shapes) in thefigures represent one example of the boundaries. One of ordinary skillin the art will appreciate that, in some examples, one element may bedesigned as multiple elements or that multiple elements may be designedas one element. In some examples, an element shown as an internalcomponent of another element may be implemented as an external componentand vice versa. Furthermore, elements may not be drawn to scale.

FIG. 1 illustrates a schematic overview of a digital pathology workflowto predict heart failure.

FIG. 2 illustrates an example deep learning architecture for heartfailure classification.

FIG. 3 illustrates receiver operator characteristic (ROC) curves fordetection of clinical heart failure or severe tissue pathology

FIG. 4 illustrates image-level and patient-level results for differentapproaches to detecting clinical heart failure or severe tissuepathology.

FIG. 5 illustrates an example apparatus that predicts heart failure.

FIG. 6 illustrates an example method for predicting heart failure.

FIG. 7 illustrates an example method for predicting heart failure.

FIG. 8 illustrates an example method for training a convolutional neuralnetwork to predict heart failure.

FIG. 9 illustrates an example computer in which example methods andapparatus described herein may operate.

DETAILED DESCRIPTION

Embodiments described herein train and employ a deep learningconvolutional neural network (CNN) classifier to predict clinical heartfailure from digitized hematoxylin and eosin (H&E) stained whole slideimages (WSIs) of heart tissue. Embodiments described herein facilitatethe automatic identification of cardiovascular pathology from WSIs ofcardiovascular histopathology, including tissue derived from hearttransplants, explants, surgical tissue samples, EMBs, or otherdiagnostic cardiac biopsies. Quantitative histomorphometry involves theconversion of a digitized histopathology slide into a series ofquantitative measurements of tissue morphology. Example embodimentsfacilitate directly transforming a digitized WSI of heart tissue into aprobability of a patient-level diagnosis. Example methods and apparatusautomatically and reproducibly quantify the extent of cardiovascularpathology in a WSI, whether due to ischemic causes, non-ischemic causes,or other causes. Example methods and apparatus use an automatedcomputational histomorphometric image analysis to facilitate apatient-level prediction of clinical heart failure or heart diseasebased on sub-visual features extracted from cardiac histopathology WSIswith an accuracy of at least 96.2%. Example methods and apparatus aremore accurate in predicting heart failure than conventionalfeature-engineering approaches, and are also more accurate than experthuman pathologists. Example methods and apparatus facilitate automatedand reproducible classification of WSIs because the CNN is deterministicand will repeatedly produce the same classification on the same inputsample, in contrast to human experts that exhibit inter-expert andintra-expert variances. Automated analysis and grading of cardiachistopathology as described herein may further be employed as anobjective second read of EMBs to improve heart disease characterizationand detection. Example embodiments detect tissue-level pathology, andfacilitate the timely re-examination of clinically normal patients whomay subsequently be found to have severe tissue pathology that wasotherwise undetected by conventional approaches. Embodiments describedherein may further predict immune-mediated transplant rejection based,at least in part on a CNN's analysis of cardiovascular histopathology.Example embodiments facilitate deep-phenotyping to support precisionmedicine initiatives to enhance the targeting of therapeutics based onthe deeper understanding of disease mechanisms and their manifestationswithin individual patients.

Conventionally, deep learning CNNs have not been applied to imageanalysis of cardiac histopathology represented in WSIs. Deep learning isan example of representation learning, which is a class of machinelearning approaches in which discriminative features are not engineeredor pre-specified, but are instead learned by the machine learningclassifier directly from raw data. In a typical CNN, there are multipleartificial neurons, or nodes (also referred to as parameters), arrangedin a hierarchical network of successive convolutional, max-pooling, andfully-connected layers. The hierarchical network structure facilitatesthe CNN model to approximate complex functions and learn non-linearfeature combinations that maximally discriminate among the classes. Theapproximation of complex functions may occur as a result of each layerbeing a higher level abstraction than the previous layer, where, forexample, an earlier layer learns pixel-level features, while laterlevels combine the lower level features into more complex objects orconcepts. When a CNN model is trained on a sufficiently large data set,the CNN model may generalize to unseen examples from a population.

Embodiments described herein employ a modified deep learning CNNclassifier that provides superior performance to conventional approachesthat employ human engineered features in a WND-CHARM+RF pipeline or evenconventional CNNs. As used herein, WND-CHARM refers to “WeightedNeighbor Distances using a Compound Hierarchy of Algorithms RepresentingMorphology”. RF refers to a random forests classifier. Example methodsand apparatus provide superior performance than conventional approachesdue, at least in part, to the learning of novel discriminative featuresor nonlinear feature combinations that are not present in theWND-CHARM+RF pipeline, and to the improved efficiency and performance ofthe modified CNN architecture described herein.

In one embodiment, a population includes 209 patients divided into twocohorts. A first cohort includes patients with end-stage heart failure(Failing or Fal, N=96) and a second cohort includes a comparison groupof patients without known heart failure (Non-Failing or NF, N=113).Tissue from the failing cohort is obtained at the time of cardiacexplantation for transplant, or as core samples obtained at the time ofleft ventricular assist device (LVAD) implantation. The failing cohortincludes patients with clinically diagnosed ischemic cardiomyopathy(ICM) or idiopathic dilated cardiomyopathy (NICM). Organ donors withouta history of heart failure comprise the NF cohort. In this embodiment,WSIs are generated by H&E staining formalin-fixed, paraffin-embeddedtransmural left ventricular free wall sections from each heart thendigitizing the sections at 20× magnification. WSIs may be acquired, inone example, using an Aperio ScanScope slide scanner, or other WSIscanner. The images are down-sampled to 5× magnification for imageanalysis. Down-sampling to 5× magnification facilitates expert humanassessment of tissue pathology. For example, an apparent magnificationof 5× facilitates the identification by expert pathologists ofmacroscopic (tissue level) and microscopic (cellular level) pathology ina given ROI. An apparent magnification of 5× further facilitatesefficient automated image analysis.

In this embodiment, the population of patients is randomly split intotwo datasets: a first cohort of 104 patients was designated fortraining, and a separate cohort of 105 patients was held out as anindependent test set. The training dataset was further split at thepatient level into three-folds for cross-validation to assess trainingand validate parameters or properties of the CNN. For a patient's WSIimage in both datasets, eleven non-overlapping regions of interest (ROI)were extracted randomly from with the tissue border of the WSI. An ROIin this example has an area of 2500 μm². Randomly extractingnon-overlapping ROIs from the tissue area facilitates the sampling ofthe extent of disease present throughout the tissue sample. In thisembodiment, eleven non-overlapping ROIs are used so that a voting-basedscheme will yield a majority vote with no ties for a binary classifier.Thus, an individual ROI will yield an image-level prediction (e.g.,disease v. no disease, failing v. non-failing) and a final,patient-level prediction may be determined by a majority prediction(e.g. majority vote) from the individual ROIs. Stain normalization maybe applied to a WSI and ROIs may be extracted before stain normalization(raw) and after stain normalization (normalized) to assess the need forstain normalization in cardiac histopathology. Stain normalization maybe, for example, Macenko stain normalization or other type of stainnormalization. In a preferred embodiment, no stain normalization isemployed. The set of training images are used to build independentclassifiers to predict heart failure. In a preferred embodiment, theindependent classifier is built using a deep learning approach thatrequires no image segmentation. For testing and comparison purposes, aWND-CHARM classifier that uses conventional feature engineering coupledwith a random decision forest classifier was built.

FIG. 1 is a schematic overview of a digital pathology workflow topredict heart failure suitable for use by embodiments described herein.At 110, a set of patients are divided into a training dataset and a testdataset. WSIs 112 acquired from the set of patients are scanned and ROIs114 are extracted for image analysis. In this example, all ROIs from thesame patient were given the same label which was determined by whetherthe patient had clinical or pathological evidence of heart disease. Forexample, an ROI may be labeled as “disease or no disease”, or “failingor non failing”. FIG. 1 illustrates a workflow that includes twodifferent approaches 120 to model training. FIG. 1 illustrates, at afirst point 122, training a conventional engineered feature classifier,for example a WND-CHARM+RF classifier. FIG. 1 further illustrates, at asecond point 124, training a naïve deep learning CNN classifier asclaimed and described herein. Three-fold cross validation was used totrain and test both the conventional heart failure classifier 122 andthe preferred deep learning CNN embodiment 124. The trained models werethen evaluated at 130 at both the image level and the patient-level on aheld-out test dataset. In this example, a probability of failurep(Fal)>0.5 results in a classification of failure.

A conventional feature engineering approach may employ, for example,WND-CHARM+RF to compute image features for a patient ROI. In thisexample, the WND-CHARM-based conventional approach computed 4059 imagefeatures for each patient ROI. The top 20 WND-CHARM features were thenselected using a minimum redundancy, maximum relevance approach, and thetop 20 most discriminative features were then input to a random decisionforest (RF) classifier. The RF classifier was used to calculate aper-ROI (e.g. image-level) probability of heart failure, which wasthresholded at 0.5. A per-ROI probability of heart failure greater than0.5 indicated failure, while a per-ROI probability of heart failure lessthan 0.5 indicated non-failure. The fraction of ROIs as failing gave thepatient-level probabilities.

Example embodiments improve on conventional approaches by using deeplearning to predict clinical heart failure using only the input imageswithout requiring feature crafting. In one embodiment, the deep learningmodel uses a modified AlexNet architecture. A conventional AlexNet CIFAR10 architecture accepts a 32×32 pixel input. Other AlexNetconfigurations accept input sizes of 225 pixels by 225 pixels. Exampleembodiments employ a modified network that accepts 64×64 pixel RGB imagepatches (having an area of 128 μm²) with a label corresponding to thecohort to which the patient from which the image patch was acquiredbelongs. (e.g. failing, non-failing). A 64×64 pixel input size providesan optimized balance of speed and accuracy compared to a 32×32 pixelapproach which is less accurate, and compared to input sizes greaterthan 64×64 pixels that do not significantly improve accuracy butincrease the time required to process.

Example embodiments further reduce the number of nodes (also referred toas parameters or neurons) employed in the network while achievingsimilarly accurate performance but with a significant reduction intraining time compared to conventional approaches. For example, aconventional AlexNet CIFAR CNN architecture employs over 145 thousandparameters (e.g. nodes, neurons) while example embodiments employapproximately 13 thousand parameters, which is nearly an order ofmagnitude reduction in the number of neurons or parameters employed.Reducing the number of nodes reduces the training time required to trainthe CNN compared to conventional approaches, and further reduces thecomputational complexity of analyzing an image with the CNN, which inturn reduces the energy and computational resources required to operatethe CNN or a CADx system that employs the CNN. For example, embodimentsdescribed herein reduce the amount of data required to train the system.The larger the number of parameters, the larger the number ofpatients/samples required before the system can be trained togeneralized well. This is problematic in a medical domain where it isimpractical or difficult, if indeed possible, to acquire the millions ofexemplars that may be obtained from social media in the facialrecognition or language detection domains. By reducing the amount ofdata required to train the CNN, example embodiments thus improve onconventional approaches.

Additionally, example embodiments employ a fully-convolutional network,in which the max-pooling layers and fully connected layers employed inconventional neural networks are replaced by convolutional layers,facilitating image-level predictions (e.g. producing output images)significantly faster than conventional approaches. In one embodiment,the deep learning classifier is trained using one-hundred patches perROI, per patient. The training set is further augmented by rotating eachpatch by 90 degrees. Embodiments may apply additional numbers ofrotations, or rotations of different numbers of degrees, to patches. Inthis example, each fold of the three-fold cross validation was trainedusing NVIDIA DIGITS and Caffe for 30 epochs on a Titan X GPU, with CUDA7.5 and cuDNN, and optimized by stochastic gradient descent with a fixedbatch size of 64. DIGITS facilitates the viewing of results, while Caffeis employed for processing. Embodiments described herein may conductimage processing and analysis using, for example, MATLAB version R2015or newer, Python, or Caffe, or may employ other image processing andanalysis packages. While 30 epochs are used in this example, othernumbers of epochs may be employed.

FIG. 2 illustrates an example embodiment of a deep learning architecture200 for heart failure classification that may be employed by exampleembodiments. Deep learning architecture 200 is suitable use with methodsand apparatus described herein, including apparatus 500, method 600,method 700, or method 800. An image 210 of an ROI is accessed. The image210 is, in this example, a digitized H&E WSI of an ROI of heart tissue.The digitized WSI may be a red-green-blue (RGB) WSI. In anotherembodiment, the digitized WSI may be a grayscale WSI, another colorspace (e.g., HUV) WSI, or other type of WSI. A plurality of patches areextracted 220 from the image 210. In this example, only one patch 222 isillustrated for clarity. Recall that in one embodiment, one-hundredpatches are extracted from an ROI. By sub-sampling one hundred 64×64pixel patches from an ROI, example embodiments augment the data used bythe CNN by increasing the diversity of inputs presented to the CNN. Forexample, a given 256 by 256 pixel image with disease should produce a“disease” prediction (e.g. failure) regardless of where the 64 by 64pixel sub-patch is selected from within the image. By randomlysub-sampling patches from an ROI, example embodiments improve onconventional approaches by both increasing the number N of inputsavailable to the CNN, while also training the CNN to be insensitive tominor image transforms (e.g., translations). In other embodiments, othernumbers of patches greater than one-hundred or less than one-hundred maybe extracted. Sampling fewer patches may not represent the entire areafrom an ROI. Sampling more patches introduces redundancies in whichpatches overlap and become highly correlated (e.g. the same patchshifted by one pixel), and more patches will further prolong trainingtime. In one embodiment, the number of patches extracted from the ROImay be user adjustable. In one embodiment, patches are rotated ninetydegrees to further augment the training data set. In another embodiment,patches may be rotated other numbers of degrees, for example, onedegree, two degrees, 45 degrees, 180 degrees, or 270 degrees.

Patch 222 is provided as input to a deep learning CNN 230. Embodimentsdescribed herein may employ a deep learning CNN that includesconvolutional layers 231-237. In this example, each convolutional layerof CNN 230 contains a rectified linear activation unit (ReLu) as well asbatch normalization. Batch normalization corrects covariate shit betweenlayers of the network. Covariate shift may be defined as a change in thedistribution of a function's domain. Covariate shift may complicate thetraining of a CNN. In this embodiment, the CNN is composed ofalternating convolutional, activation, and batch normalization layers.In one embodiment, an activation layer may include a ReLu through whicha weighted sum of the inputs from the previous layer may be passed. TheReLu is a non-linear function. In another embodiment, the activationlayer may employ other, different non-linear functions. Table 1 belowdescribes one embodiment of a CNN suitable for use by embodimentsdescribed herein.

TABLE 1 Number Of Layer Kernel Stride Kernels Input 64 × 64 Conv1a 3 116 Bn1a Relu1a Conv1b 2 2 16 Bn1b Relu1b Conv2a 3 1 16 Bn2a Relu2aConv2b 3 2 16 Bn2b Relu2b Conv3a 3 1 16 Bn3a Relu3a Conv3a 4 2 16 Bn3aRelu3a Fc-8-conv 5 2

The CNN described by table 1 includes seven layers. In this example,input includes at least one 64 pixel by 64 pixel patch. The 64 pixel by64 pixel patch may be extracted from an RGB image, a grayscale image, ahue-saturation-value (HSV) image, a color deconvoluted image, or animmunohistochemistry (IHC) image, including a WSI. A first convolutionallayer (conv1a) has a kernel of size 3, a stride of 1, and an output of16 kernels (3, 1, 16). The kernel value indicates the number of filtersat that layer. The stride indicates how the kernel convolves about theinput: a stride of one indicates a one-pixel shift. Subsequentconvolutional layers include conv1b(2, 2, 16), conv2a(3, 1, 16),conv2b(3, 2, 16), conv3a(3, 1, 16), and conv3b(4, 2, 16). The fullyconnected layer Fc-8-conv has a stride of size 5 and an output of 2. Thefully connected layer Fc-8-conv's output of 2 indicates a probabilitythat the input is failing or non-failing. The fully connected layerFc-8-conv may be a convolutional layer having a kernel exactly the sizeof the previous layer. Fully connected layer Fc-8-conv thus acts like afully connected layer, but has improved numerical properties. In otherembodiments, other numbers of layers may be employed. In otherembodiments, layers may have different kernels, different strides, anddifferent outputs.

CNN 230 produces an output 240 that represents a probability that thepatch 222 is a failing patch. FIG. 2 further illustrates an example setof training patches 280. The set of training patches 280 includes a setof failing patches 282, and a set of non-failing patches 284. The set oftraining patches 280 is used to train or test the deep learning CNN 230.

Embodiments described herein train a CNN using k-cross validation.Cross-validation is a technique to measure how a classifier willgeneralize outside the training dataset. The original training data setis split into k non-overlapping groups of patients, where k is aninteger. In one embodiment, the original training data set is split intothree non-overlapping groups of patients. The first k-1 groups (in thisexample, two) are used to train the CNN and the remaining group is usedto test to CNN. No patient is ever in the training group and the testgroup at the same time. This process is repeated k times until all thepatients in the original training data set have been used for trainingand testing, but never at the same time. In one embodiment, training theCNN may include using backpropagation to compute the gradient of anobjective function. In another embodiment, other techniques for findingthe minima of the objective function may be employed.

Embodiments described herein may detect tissue pathology even inpatients without diagnosed pre-existing heart failure or pathology. Forexample, a patient without clinically diagnosed pre-existing heartfailure may be predicted as failing with a very high probability by aCNN at both the image-level and at the patient-level. In this example,the non-failing patient, while without heart failure, has severe oroccult tissue pathology that was undetected by conventional diagnostictechniques. The severe tissue pathology may be predictive of futureheart failure. Thus, embodiments described herein may classify a patientas either non-failing or “abnormal or failing”. Classifying a patient as“failure/abnormal” that may otherwise be deemed a false positiveprovides the additional benefit of directing resources in a timelymanner to patients that may go untreated until conventionally detectablesymptoms develop.

FIG. 3 illustrates graphs of the receiver operator characteristic (ROC)curves for the detection of clinical heart failure or severe tissuepathology. The vertical axes represent the true positive rate, and thehorizontal axes represent the false positive rate. In graphs 310, 320,330, and 340, the solid line represents the deep learning ROC curve, thebarred line represents the conventional WND-CHARM+RF ROC curve, and thedashed line represents chance or mere guessing.

Graph 310 illustrates the ROC curve for image-level detection on atraining data set. Graph 310 illustrates the ROCs of DL embodimentsdescribed herein vs. conventional RF approaches. In graph 310, p<0.0001,using a two-sample Kolmogorov-Smirnov (KS) test, where p represents thep-value.

Graph 320 illustrates the ROC curves for patient-level detection on thetraining data set. Graph 320 represents the ROCs of deep learningembodiments described herein (DL) vs. conventional WND-CHARM+RF (RF)approaches, also using a two-sample KS test.

Graph 330 represents the ROC of deep learning embodiments describedherein (DL) vs. conventional WND-CHARM+RF (RF) approaches forimage-level detection on a held-out test data set. In graph 330,p<0.0001, and a two-sample KS test was also used.

Graph 340 represents the ROC of deep learning embodiments describedherein (DL) vs. conventional WND-CHARM+RF (RF) approaches forpatient-level detection on the held-out test data set. In graph 340, atwo-sample KS test was also used. FIG. 3 thus provides evidence of theimproved accuracy of example embodiments compared to conventionalapproaches for predicting heart failure.

FIG. 4 illustrates a table 400 of patient-level performance evaluationfor predicting clinical heart failure or severe tissue pathology fromH&E stained digitized WSIs for a held-out test set. The resultspresented in table 400 represent the mean +−SD of three models. Each ofthe three models was trained on 770 images acquired from 70 patients.The three models were evaluated on the held-out test set of 105patients. The patient-level diagnosis is a majority vote over all theimages (e.g. ROIs) from a single patient. Statistics presented in table400 were determined by an unpaired two-sample t-test with an N of threefolds. The “Random Forest” column indicates the results of aWND-CHARM+RF approach to predicting heart failure. The “Deep Learning”column indicates the results of the deep learning CNN approach employedby embodiments described herein. On every metric, the deep learning(e.g. CNN) approach out-performs the conventional engineered featuresrandom forest approach. For example, the AUC for image-level results forthe conventional WND-CHARM+RF approach is 0.935+−0.001, while the AUCfor image level results for example embodiments is 0.977+−0.01.Similarly, the AUC for patient-level results for the conventionalWND-CHARM+RF approach is 0.960+−0.01, while the AUC for patient-levelresults for example embodiments is 0.989+−0.01. Thus, FIG. 4 providesevidence of the improved accuracy of example embodiments compared toconventional technologies for predicting heart failure.

Example methods and apparatus thus demonstrably improve on conventionaltechnologies for predicting clinical heart failure. For example, methodsand apparatus described herein predict clinical heart failure or severetissue pathology with an average area under the curve (AUC) accuracy ofat least 0.977, compared with conventional approaches that achieve anaverage AUC of only 0.935. Example embodiments further improve on theperformance of expert human pathologists in predicting heart failure orsever tissue pathology, in that expert human pathologists typicallyachieve an individual accuracy of only 75% at the patient level with aCohen's kappa inter-rate agreement of 0.40. By increasing the accuracywith which clinical heart failure or severe tissue pathology ispredicted, example methods and apparatus produce the concrete,real-world technical effect of increasing the probability that at-riskpatients receive timely treatment tailored to the particular pathologythey exhibit. The additional technical effect of reducing theexpenditure of resources and time on patients who have a less aggressivepathology is also achieved. Example embodiments further improve onconventional approaches by providing a more accurate second reader tofacilitate the reduction of inter-reader variability among humanpathologists. Example methods and apparatus thus improve on conventionalmethods in a measurable, clinically significant way.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a memory. These algorithmic descriptions and representationsare used by those skilled in the art to convey the substance of theirwork to others. An algorithm, here and generally, is conceived to be asequence of operations that produce a result. The operations may includephysical manipulations of physical quantities. Usually, though notnecessarily, the physical quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a logic, and so on. The physicalmanipulations create a concrete, tangible, useful, real-world result.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, and so on. It should be borne in mind,however, that these and similar terms are to be associated with theappropriate physical quantities and are merely convenient labels appliedto these quantities. Unless specifically stated otherwise, it isappreciated that throughout the description, terms including processing,computing, calculating, determining, and so on, refer to actions andprocesses of a computer system, logic, processor, or similar electronicdevice that manipulates and transforms data represented as physical(electronic) quantities.

Example methods may be better appreciated with reference to flowdiagrams. While for purposes of simplicity of explanation, theillustrated methodologies are shown and described as a series of blocks,it is to be appreciated that the methodologies are not limited by theorder of the blocks, as some blocks can occur in different orders and/orconcurrently with other blocks from that shown and described. Moreover,less than all the illustrated blocks may be required to implement anexample methodology. Blocks may be combined or separated into multiplecomponents. Furthermore, additional and/or alternative methodologies canemploy additional, not illustrated blocks.

FIG. 5 illustrates an example apparatus 500 that predicts clinical heartfailure. Apparatus 500 includes a processor 510, a memory 520, aninput/output interface 530, a set of circuits 540, and an interface 550that connects the processor 510, the memory 520, the input/outputinterface 530, and the set of circuits 540. The set of circuits 540includes a pre-processing circuit 541, an image acquisition circuit 543,a deep learning circuit 545, and a classification circuit 547.

Memory 520 is configured to store a digitized whole slide image (WSI) ofa region of tissue. The region of tissue may be, for example, a sectionof tissue demonstrating heart failure pathology, and thus the WSI may beof cardiovascular histopathology. The digitized WSI may be an RGB image,a grey scale image, or other color space image. The digitized WSI has aplurality of pixels. A pixel in the digitized WSI has an intensity or acolor value. In one embodiment, the volume illustrated in the digitizedWSI represents tissue collected from a patient with clinically diagnosedischemic cardiomyopathy (ICM), or idiopathic dilated cardiomyopathy(NICM), or other heart disease related pathology. The tissue may becollected from a patient who received a heart transplant or a LVAD. Inone embodiment, the image may be acquired from an organ donor without ahistory of heart failure. The digitized WSI may represent tissue derivedfrom a transplant, an explant, a surgical tissue sample, an EMB, orother surgical or biopsy procedure. In other embodiments, the volumeillustrated in the image may be associated with other imaging systems,or be of other regions demonstrating other types of pathology.

Pre-processing circuit 541 is configured to generate a pre-processedWSI. Pre-processing circuit 541 may generate the pre-processed WSI bydownsampling the digital WSI. For example, in one embodiment, the WSI isacquired at a 20× magnification. Generating the pre-processed WSI mayinclude downsampling the digital WSI to an apparent magnification of 5×.In another embodiment, the WSI may be acquired at another, differentmagnification, or the WSI may be downsampled to another, differentapparent magnification. In another embodiment, pre-processing the WSImay include color normalization, or determining where in the slide thetissue sample is located.

Image acquisition circuit 543 is configured to randomly select a set ofnon-overlapping ROIs from the pre-processed WSI. Image acquisitioncircuit 543 provides the set of non-overlapping ROIs to the deeplearning circuit 545. In this embodiment, a member of the set ofnon-overlapping ROIs has dimensions of 256 pixels by 256 pixels. The setof non-overlapping ROIs has an odd cardinality. In one embodiment, theset of non-overlapping ROIs extracted by image acquisition circuit 543includes eleven non-overlapping ROIs. Selecting or accessing an ROI fromthe WSI includes acquiring electronic data, reading from a computerfile, receiving a computer file, reading from a computer memory, orother computerized activity.

Deep learning circuit 545 is configured to generate an image-levelprobability that a member of the set of non-overlapping ROIs is anon-failure ROI or a failure/abnormal pathology ROI. In one embodiment,deep learning circuit 545 provides the member of the set ofnon-overlapping ROIs to a CNN. Providing the member of the set ofnon-overlapping ROIs includes acquiring electronic data, reading from acomputer file, receiving a computer file, reading from a computermemory, or other computerized activity. The CNN produces the image-levelprobability based, at least in part, on the member of the set ofnon-overlapping ROIs. The CNN resolves features present in the digitizedWSI at a higher order or higher level than a human can resolve.

In one embodiment, the CNN is configured to accept one-hundred 64 pixelby 64 pixel input patches per member of the set of non-overlapping ROIs.In another embodiment, the CNN may be configured to accept another,different number of input patches, or to accept input patches havingdifferent dimensions. For example, the CNN may be configured to accept32 pixel by 32 pixel input patches, or 128 pixel by 128 pixel inputpatches.

In one embodiment, the CNN is a seven-layer CNN. In this embodiment, theCNN has 13460 parameters (e.g., neurons, nodes). In another embodiment,the CNN may have another, different number of neurons. For example, theCNN may have 12000 neurons, or 15000 neurons.

In this embodiment, the CNN includes a first layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3 and uses a stride of 1. In this embodiment, the kernel is a 3×3matrix, with the matrix values representing the learned weights.

In this embodiment, the CNN also includes a second layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 2, and uses a stride of 2.

In this embodiment, the CNN also includes a third layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3, with a stride of 1.

In this embodiment, the CNN also includes a fourth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3, with a stride of 2.

In this embodiment, the CNN also includes a fifth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3, with a stride of 1.

In this embodiment, the CNN also includes a sixth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 4, and uses a stride of 2.

In this embodiment, the CNN further includes a fully connected layer.The fully connected layer has a kernel of 5 and an output of 2. Inanother embodiment, other numbers of layers, kernels, strides, oroutputs may be employed.

Classification circuit 547 is configured to compute a patient-levelprobability that the patient from which the region of tissue representedin the WSI was acquired has failure or non-failure. Classificationcircuit 547 computes the patient-level probability based, at least inpart, on the image-level probability. In one embodiment, theclassification circuit 547 computes the patient-level probability basedon a majority vote of the image-level probabilities associated withmembers of the set of non-overlapping ROIs. Selecting a set ofnon-overlapping ROIs that has an odd cardinality facilitates avoidingtie votes and thus achieving a majority vote. In this embodiment,apparatus 500 identifies a patient with heart failure with a sensitivityof 99% and a specificity of 93%.

In another embodiment, classification circuit 547 may control a computeraided diagnosis (CADx) system to classify the region of tissuerepresented in the WSI based, at least in part, on the probabilitygenerated by deep learning circuit 545. For example, classificationcircuit 547 may control a CADx system to distinguish the image based, atleast in part, on the probability or the classification generated byclassification circuit 547. In other embodiments, other types of CADxsystems may be controlled, including CADx systems for distinguishingother types of tissue presenting other, different pathologies that maybe distinguished based on features detected by deep learning circuit 545represented in a digitized WSI. For example, embodiments describedherein may be employed to classify or grade breast cancer (BCa) based onWSIs of BCa tissue. Other embodiments may be employed to classify kidneydisease, or brain pathologies.

In one embodiment of apparatus 500, the set of circuits 540 furtherincludes a training circuit configured to train the CNN. Training theCNN includes accessing a training dataset of WSIs. In one embodiment,the WSIs are acquired at 20× magnification. The training datasetincludes a first subset of WSIs of tissue acquired from patientsdemonstrating clinically diagnosed heart failure, and a second, disjointsubset of WSIs of tissue acquired from patients that have not beenclinically diagnosed with heart failure. In one embodiment, the firstsubset is acquired from heart failure patients receiving a hearttransplant or LVAD. Tissue acquired from the first subset may beacquired post-explant, or may be acquired as surgical core samples forpatients receiving LVAD. In this embodiment, the second subset isacquired from organ donors without a history of heart failure, but wherethe heart was not used for transplant. Members of the training datasetmay be downsampled to an apparent magnification of 5×.

The training circuit is further configured to split the training datasetinto k-1 groups, where k is an integer. In this embodiment, three-foldcross validation is employed. In other embodiments, other forms of crossvalidation may be employed by the training circuit. In one embodiment,before splitting the training dataset, a held out testing set is removedfrom the training dataset.

The training circuit is further configured to train the CNN with thefirst k-1 groups. Training the CNN includes extracting 64 pixel by 64pixel RGB image patches from an ROI randomly selected from a member ofthe first k-1 groups of WSIs, where the member has a label correspondingto the cohort to which the patient from which the WSI was acquiredbelongs (e.g. failing or non-failing). The CNN is trained usingone-hundred patches per ROI. In one embodiment, the training data set isaugmented by rotating a patch by ninety degrees. In one embodiment, theCNN is trained per fold for thirty epochs using stochastic gradientdescent with a fixed batch size of 64. In another embodiment, the CNNmay be trained for another, different number of epochs, or with another,different batch size. In one embodiment, the training circuit isconfigured to train the CNN using backpropagation.

The training circuit is further configured to test the CNN with theremaining group. In one embodiment, the training circuit may furthertest the CNN using the held-out testing set.

The training circuit is further configured to determine if all thepatients of the training dataset have been used for training the CNN andtesting the CNN. Upon determining that all the patients have been usedfor training the CNN, the training circuit is configured to end thetraining. In one embodiment, the training circuit may end the trainingupon determining that a threshold percentage or proportion of patientshave been used for training and testing the CNN. In another embodiment,the training circuit may end the training upon determining thatthreshold level of accuracy has been achieved by the CNN, or upondetermining that training progress has slowed to a threshold level.

In one embodiment of apparatus 500, the set of circuits 540 furtherincludes a display circuit. The display circuit may control the CADxsystem to display the digitized WSI or the probability that the patienthas heart failure on a computer monitor, a smartphone display, a tabletdisplay, or other displays. Displaying the digitized WSI or theprobability that the patient has heart failure may also include printingthe WSI or the probability that the patient has heart failure. Thedisplay circuit may also control the CADx to display an image of the ROIor of an input patch. The display circuit may also control the CADxsystem to display operating parameters or characteristics of the CNN,during both training and testing and clinical operation. Displaying thedigitized WSI involves changing the character of the information presentin a biopsy sample (e.g. biological tissue), to a WSI, changing theinformation present in the WSI to information in the digitized WSI, andthen to information suitable for display on, for example, a computermonitor, a smartphone display, a tablet display, or other displays.

FIG. 6 illustrates a computerized method 600 for predicting heartfailure. Method 600 includes, at 610, accessing a digitized whole slideimage (WSI) of cardiovascular histopathology. Accessing a digitized WSIincludes acquiring electronic data, reading from a computer file,receiving a computer file, reading from a computer memory, or othercomputerized activity. In one embodiment, the volume illustrated in theWSI is associated with H&E stained tissue derived from a transplant, anexplant, a surgical tissue sample, or an endomyocardial biopsy of apatient demonstrating pathology associated with heart failure. The WSImay be acquired at 20× magnification. In other embodiments, differenttypes of tissue acquired by different procedures may be imaged usingdifferent imaging techniques.

Method 600 also includes, at 620, generating a pre-processed WSI bydownsampling the WSI. In one embodiment, downsampling the WSI includesdownsampling the WSI to an apparent magnification of 5×. In anotherembodiment, downsampling the WSI may include downsampling the WSI toanother, different apparent magnification. Pre-processing the WSI mayfurther include color normalization, noise reduction, smoothing, or edgeamplification.

Method 600 also includes, at 630, extracting a set of non-overlappingregions of interest (ROIs) from the pre-processed WSI. In oneembodiment, the set of non-overlapping ROIs has an odd cardinality. Inthis embodiment, the set of non-overlapping ROIs is selected randomlyfrom the WSI. In another embodiment, the set of non-overlapping ROIs maybe selected non-randomly, or according to a pattern. In one embodiment,the set of non-overlapping ROIs includes 11 non-overlapping ROIs. Inanother embodiment, the set of non-overlapping ROIs includes another,different number of non-overlapping ROIs. In one embodiment, a member ofthe set of non-overlapping ROIs has dimensions of 256 pixels by 256pixels. In another embodiment, a member of the set of non-overlappingROIs may have other, different dimensions. The ROI dimensions may beuser adjustable.

Method 600 also includes, at 640, providing the set of non-overlappingROIs to a deep learning convolutional neural network (CNN). Providingthe set of non-overlapping ROIs includes acquiring electronic data,reading from a computer file, receiving a computer file, reading from acomputer memory, or other computerized activity. In one embodiment, theCNN accepts one-hundred 64 pixel by 64 pixel input patches per member ofthe set of non-overlapping ROIs. In another embodiment, another,different number of input patches per member of the set ofnon-overlapping ROIs may be provided to the CNN. In another embodiment,the input patches may have different dimensions. For example, an inputpatch may have dimensions of 32 pixels by 32 pixels, or 128 pixels by128 pixels. In one embodiment, input patch dimensions may be userselectable.

In one embodiment, the CNN is a seven-layer CNN. The CNN, including aseven layer CNN, has alternating convolutional, activation, batchnormalization, and convolutional fully connected layers. For example, inone embodiment, the CNN includes a first layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has a kernel of 3, astride of 1, and an output of 16.

In this embodiment, the CNN includes a first layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has a has 16 kernelsof size 3 and uses a stride of 1.

In this embodiment, the CNN also includes a second layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 2, and uses a stride of 2.

In this embodiment, the CNN also includes a third layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3, with a stride of 1.

In this embodiment, the CNN also includes a fourth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 3, with a stride of 2.

In this embodiment, the CNN also includes a fifth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has a 16 kernels ofsize 3, with a stride of 1.

In this embodiment, the CNN also includes a sixth layer comprising aconvolutional layer, a batch normalization layer, and an activationlayer. In this embodiment, the convolutional layer has 16 kernels ofsize 4, and uses a stride of 2.

In this embodiment, the CNN further includes a fully connected layer.The fully connected layer has a kernel of 5 and an output of 2. Inanother embodiment, the CNN may have other numbers of layers. In anotherembodiment, other kernels, strides, or outputs may be employed.

Method 600 also includes, at 650, receiving, from the CNN, a probabilitythat a member of the set of non-overlapping ROIs is a non-failure ROI,or a failure/abnormal pathology ROI. In one embodiment, an ROI with aprobability of failure p(fal)>0.50 is classified as failing.

Method 600 further includes, at 660, controlling a computer assisteddiagnosis (CADx) system to classify the region of tissue represented inthe WSI as a non-failure histopathology or as a failure/abnormalpathology histopathology based, at least in part, on the probabilitiesassociated with members of the set of non-overlapping ROIs. In oneembodiment, the CADx system classifies the region of tissue based on amajority vote of the probabilities associated with members of the set ofnon-overlapping ROIs. For example, in an embodiment in which eleven ROIsare provided to the CNN, if the CNN returns probabilities such that tenof the eleven ROIs are classified as failure/abnormal, the majority votewould indicate that the region of tissue is failure/abnormal.

Improved identification of patients with heart failure using deeplearning CNNs with a sensitivity of 99% and a specificity of 93% mayproduce the technical effect of improving treatment efficacy byincreasing the accuracy of and decreasing the time required to identifypatients with heart failure. Treatments and resources may be moreaccurately tailored to patients with a likelihood of benefiting fromsaid treatments and resources, so that more appropriate treatmentprotocols may be employed.

Using a more appropriately modulated treatment may lead to lessaggressive therapeutics being required for a patient or may lead toavoiding or delaying a biopsy, a resection, or other invasive procedure.When patients experiencing heart failure are more quickly and moreaccurately distinguished, patients most at risk may receive a higherproportion of scarce resources (e.g., therapeutics, physician time andattention, hospital beds) while those less at risk may be sparedunnecessary treatment, which in turn spares unnecessary expenditures andresource consumption. Example methods, apparatus, and other embodimentsmay thus have the additional effect of improving patient outcomescompared to conventional approaches.

While FIG. 6 illustrates various actions occurring in serial, it is tobe appreciated that various actions illustrated in FIG. 6 could occursubstantially in parallel. By way of illustration, a first process couldinvolve pre-processing a WSI, a second process could involve extractinga set of non-overlapping ROIs, and a third process could involveproviding ROIs to a CNN. While three processes are described, it is tobe appreciated that a greater or lesser number of processes could beemployed and that lightweight processes, regular processes, threads, andother approaches could be employed.

FIG. 7 illustrates an example method 700 that is similar to method 600but includes additional elements. Method 700 includes the elements ofmethod 600, but also includes, at 710, training the CNN.

FIG. 8 illustrates an example method 800 for training a CNN employableby embodiments described herein, including apparatus 500 or method 700.Method 800 includes, at 810, accessing a training dataset of WSIs. Inone embodiment, the WSIs are acquired at 20× magnification. The trainingdataset includes a first subset of WSIs of H&E stained tissue acquiredfrom patients demonstrating clinically diagnosed heart failure, and asecond, disjoint subset of WSIs of tissue acquired from patients thathave not been clinically diagnosed with heart failure. In oneembodiment, the first subset is acquired from heart failure patientsreceiving a heart transplant or LVAD. Tissue acquired from the firstsubset may be acquired post-explant, or may be acquired as surgical coresamples for patients receiving LVAD. In this embodiment, the secondsubset is acquired from organ donors without a history of heart failure,but where the heart was not used for transplant. Members of the trainingdataset may be downsampled to an apparent magnification of 5×. Accessingthe training dataset of WSIs includes acquiring electronic data, readingfrom a computer file, receiving a computer file, reading from a computermemory, or other computerized activity.

Method 800 also includes, at 820, splitting the training dataset intok-1 groups, where k is an integer. In this embodiment, three-fold crossvalidation is employed. In other embodiments, other forms of crossvalidation may be employed.

Method 800 also includes, at 830, training the CNN with the first k-1groups. Training the CNN includes extracting 64 pixel by 64 pixel RGBimage patches from an ROI randomly selected from a member of the firstk-1 groups of WSIs, where a patch has a label corresponding to thecohort to which the patient belongs (e.g. failing or non-failing). TheCNN is trained using one-hundred patches per ROI. In one embodiment, thetraining data set is augmented by rotating a patch by ninety degrees. Inanother embodiment, a patch may be rotated another, different number ofdegrees. In one embodiment, the CNN is trained per fold for thirtyepochs using stochastic gradient descent with a fixed batch size of 64.In another embodiment, the CNN may be trained for another, differentnumber of epochs, or with another, different batch size.

Method 800 also includes, at 840, testing the CNN with the remaininggroup. In one embodiment, the CNN may be further tested using a held-outtest group. In another embodiment, other groups may be used to test theCNN.

Method 800 further includes, at 850 determining if all the patients ofthe training dataset have been used for training the CNN and testing theCNN. Upon determining that all the patients have been used for trainingthe CNN, the training is terminated at 860. In another embodiment,training may be terminated upon meeting another, different condition.

In one example, a method may be implemented as computer executableinstructions. Thus, in one example, a computer-readable storage mediummay store computer executable instructions that if executed by a machine(e.g., computer) cause the machine to perform methods described orclaimed herein including method 600, method 700, and method 800. Whileexecutable instructions associated with the listed methods are describedas being stored on a computer-readable storage medium, it is to beappreciated that executable instructions associated with other examplemethods described or claimed herein may also be stored on acomputer-readable storage medium. In different embodiments the examplemethods described herein may be triggered in different ways. In oneembodiment, a method may be triggered manually by a user. In anotherexample, a method may be triggered automatically.

FIG. 9 illustrates an example computer 900 in which example methodsillustrated herein can operate and in which example methods, apparatus,circuits or logics may be implemented. In different examples, computer900 may be part of a digital WSI system, may be operably connectable toa digital WSI system, or may be part of a CADx system.

Computer 900 includes a processor 902, a memory 904, and input/output(I/O) ports 910 operably connected by a bus 908. In one example,computer 900 may include a set of logics 930 that perform a method ofidentifying heart failure in patients using a deep learning CNN. Thus,the set of logics 930, whether implemented in computer 900 as hardware,firmware, software, and/or a combination thereof may provide means(e.g., hardware, firmware, circuits) for identifying or predicting heartfailure or abnormal tissue pathology in a patient using WSIs of tissueacquired from the patient, and a CNN. In different examples, the set oflogics 930 may be permanently and/or removably attached to computer 900.

Processor 902 can be a variety of various processors including dualmicroprocessor and other multi-processor architectures. Processor 902may be configured to perform steps of methods claimed and describedherein. Memory 904 can include volatile memory and/or non-volatilememory. A disk 906 may be operably connected to computer 900 via, forexample, an input/output interface (e.g., card, device) 918 and aninput/output port 910. Disk 906 may include, but is not limited to,devices like a magnetic disk drive, a tape drive, a Zip drive, a flashmemory card, or a memory stick. Furthermore, disk 906 may includeoptical drives like a CD-ROM or a digital video ROM drive (DVD ROM).Memory 904 can store processes 914 or data 917, for example. Disk 906 ormemory 904 can store an operating system that controls and allocatesresources of computer 500.

Bus 908 can be a single internal bus interconnect architecture or otherbus or mesh architectures. While a single bus is illustrated, it is tobe appreciated that computer 900 may communicate with various devices,circuits, logics, and peripherals using other buses that are notillustrated (e.g., PCIE, SATA, Infiniband, 1394, USB, Ethernet).

Computer 900 may interact with input/output devices via I/O interfaces918 and input/output ports 910. Input/output devices can include, butare not limited to, digital whole slide scanners, an optical microscope,a keyboard, a microphone, a pointing and selection device, cameras,video cards, displays, disk 906, network devices 920, or other devices.Input/output ports 910 can include but are not limited to, serial ports,parallel ports, or USB ports.

Computer 900 may operate in a network environment and thus may beconnected to network devices 920 via I/O interfaces 918 or I/O ports910. Through the network devices 920, computer 900 may interact with anetwork. Through the network, computer 900 may be logically connected toremote computers. The networks with which computer 900 may interactinclude, but are not limited to, a local area network (LAN), a wide areanetwork (WAN), or other networks, including the cloud.

Examples herein can include subject matter such as an apparatus, a CADxsystem, a processor, a system, a method, means for performing acts orblocks of the method, at least one machine-readable medium includingexecutable instructions that, when performed by a machine (e.g., aprocessor with memory, an application-specific integrated circuit(ASIC), a field programmable gate array (FPGA), or the like) cause themachine to perform acts of the method or of an apparatus or system forpredicting heart failure according to embodiments and examplesdescribed.

One example embodiment includes a computer-readable storage devicestoring computer-executable instructions that, in response to execution,cause a computer assisted diagnosis (CADx) system or processor toperform operations. The operations include accessing a digitized WSI ofH&E stained cardiovascular histopathology acquired at 20× magnification.The digitized WSI may be associated with a patient.

The operations further include generating a pre-processed WSI bydownsampling the WSI to 5× apparent magnification.

The operations further include extracting a set of elevennon-overlapping regions of interest (ROIs) from the pre-processed WSI.In this embodiment, a member of the set of eleven non-overlapping ROIshas dimensions of 256 pixels by 256 pixels.

The operations further include providing the set of elevennon-overlapping ROIs to an unsupervised deep learning CNN configured toaccept one-hundred 64 pixel by 64 pixel patches from a member of the setof eleven non-overlapping ROIs. In this embodiment, the CNN is aseven-layer CNN having less than 14000 (fourteen thousand) neurons. Inthis embodiment, the CNN is trained using three-fold cross validationusing a training dataset of WSIs of left ventricular tissue acquiredfrom at least 200 patients. The at least 200 patients include a firstcohort diagnosed with end-stage heart failure, and a second, differentcohort without heart failure. A WSI associated with the first cohort islabeled as failure/abnormal, and a WSI associated with the second cohortis labeled as non-failure. In one embodiment, the CNN is further testedon a held-out testing dataset.

The operations further include receiving, from the CNN, an image-levelprobability that a member of the set of eleven non-overlapping ROIs is afailure/abnormal pathology ROI.

The operations further include controlling a CADx system or processor tocompute a patient-level probability that the region of tissuerepresented in the WSI is non-failure failure/abnormal histopathologybased. The patient-level probability is based, at least in part, on amajority vote of the probabilities associated with members of the set ofnon-overlapping ROIs. The operations may further include controlling aCADx system or a processor to predict clinical heart failure in thepatient associated with the digitized WSI based, at least in part, onthe majority vote.

References to “one embodiment”, “an embodiment”, “one example”, and “anexample” indicate that the embodiment(s) or example(s) so described mayinclude a particular feature, structure, characteristic, property,element, or limitation, but that not every embodiment or examplenecessarily includes that particular feature, structure, characteristic,property, element or limitation. Furthermore, repeated use of the phrase“in one embodiment” does not necessarily refer to the same embodiment,though it may.

“Computer-readable storage device”, as used herein, refers to a devicethat stores instructions or data. “Computer-readable storage device”does not refer to propagated signals. A computer-readable storage devicemay take forms, including, but not limited to, non-volatile media, andvolatile media. Non-volatile media may include, for example, opticaldisks, magnetic disks, tapes, and other media. Volatile media mayinclude, for example, semiconductor memories, dynamic memory, and othermedia. Common forms of a computer-readable storage device may include,but are not limited to, a floppy disk, a flexible disk, a hard disk, amagnetic tape, other magnetic medium, an application specific integratedcircuit (ASIC), a compact disk (CD), other optical medium, a randomaccess memory (RAM), a read only memory (ROM), a memory chip or card, amemory stick, and other media from which a computer, a processor orother electronic device can read.

“Circuit”, as used herein, includes but is not limited to hardware,firmware, software in execution on a machine, or combinations of each toperform a function(s) or an action(s), or to cause a function or actionfrom another logic, method, or system. A circuit may include a softwarecontrolled microprocessor, a discrete logic (e.g., ASIC), an analogcircuit, a digital circuit, a programmed logic device, a memory devicecontaining instructions, and other physical devices. A circuit mayinclude one or more gates, combinations of gates, or other circuitcomponents. Where multiple logical circuits are described, it may bepossible to incorporate the multiple logical circuits into one physicalcircuit. Similarly, where a single logical circuit is described, it maybe possible to distribute that single logical circuit between multiplephysical circuits.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim.

Throughout this specification and the claims that follow, unless thecontext requires otherwise, the words ‘comprise’ and ‘include’ andvariations such as ‘comprising’ and ‘including’ will be understood to beterms of inclusion and not exclusion. For example, when such terms areused to refer to a stated integer or group of integers, such terms donot imply the exclusion of any other integer or group of integers.

To the extent that the term “or” is employed in the detailed descriptionor claims (e.g., A or B) it is intended to mean “A or B or both”. Whenthe applicants intend to indicate “only A or B but not both” then theterm “only A or B but not both” will be employed. Thus, use of the term“or” herein is the inclusive, and not the exclusive use. See, Bryan A.Garner, A Dictionary of Modern Legal Usage 624 (2d. Ed. 1995).

While example systems, methods, and other embodiments have beenillustrated by describing examples, and while the examples have beendescribed in considerable detail, it is not the intention of theapplicants to restrict or in any way limit the scope of the appendedclaims to such detail. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing the systems, methods, and other embodiments described herein.Therefore, the invention is not limited to the specific details, therepresentative apparatus, and illustrative examples shown and described.Thus, this application is intended to embrace alterations,modifications, and variations that fall within the scope of the appendedclaims.

What is claimed is:
 1. An apparatus for predicting heart failure, theapparatus comprising: a processor; a memory that stores a digital wholeslide image (WSI) of a region of tissue derived from a heart transplant,an explant, a surgical tissue sample, or an endomyocardial biopsy, wherethe digital WSI has a plurality of pixels, and where a pixel has anintensity or a red-green-blue (RGB) color value; an input/output (I/O)interface; a set of circuits comprising a pre-processing circuit, animage acquisition circuit, a deep learning circuit, and a classificationcircuit; and an interface to connect the processor, the memory, the I/Ointerface and the set of circuits: where the pre-processing circuit isconfigured to generate a pre-processed WSI by downsampling the digitalWSI; where the image acquisition circuit is configured to randomlyselect a set of non-overlapping regions of interest (ROI)s from thepre-processed WSI, and configured to provide the set of non-overlappingROIs to the deep learning circuit, where the set of non-overlapping ROIshas an odd cardinality; where the deep learning circuit is configured togenerate an image-level probability that a member of the set ofnon-overlapping ROIs is a failure/abnormal pathology ROI, where the deeplearning circuit provides the member of the set of non-overlapping ROIsto a convolutional neural network (CNN), and where the CNN produces theimage-level probability based, at least in part, on the member of theset of non-overlapping ROIs; and where the classification circuit isconfigured to generate a patient-level probability that the patient fromwhich the region of tissue represented in the WSI was acquired isexperiencing failure or non-failure based, at least in part, on theimage-level probability.
 2. The apparatus of claim 1, where the WSI isan hematoxylin and eosin (H&E) stained WSI acquired at a 20×magnification.
 3. The apparatus of claim 2, where the pre-processingcircuit downsamples the WSI to an apparent magnification of 5×.
 4. Theapparatus of claim 1, where the set of non-overlapping ROIs extracted bythe image acquisition circuit includes eleven non-overlapping ROIs, andwhere a member of the set of non-overlapping ROIs has dimensions of 256pixels by 256 pixels.
 5. The apparatus of claim 1, where the deeplearning circuit includes a CNN configured to accept one-hundred 64pixel by 64 pixel input patches per member of the set of non-overlappingROIs.
 6. The apparatus of claim 5, where the CNN is a seven-layer CNNcomprising: a first layer comprising a convolutional layer, a batchnormalization layer, and an activation layer, the convolutional layerhaving 16 kernels of size 3, with a stride of 1; a second layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 2,and a stride of 2; a third layer comprising a convolutional layer, abatch normalization layer, and an activation layer, the convolutionallayer having 16 kernels of size 3, and a stride of 1; a fourth layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 3,and a stride of 2; a fifth layer comprising a convolutional layer, abatch normalization layer, and an activation layer, the convolutionallayer having 16 kernels of size 3, and a stride of 1; a sixth layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 4,and a stride of 2; and a fully connected layer having a 2 kernels ofsize
 5. 7. The apparatus of claim 5, where the classification circuitgenerates the patient-level probability based on a majority vote ofimage-level probabilities associated with members of the set ofnon-overlapping ROIs.
 8. The apparatus of claim 7, further comprising atraining circuit configured to train the CNN, where training the CNNincludes: accessing a training dataset, where the training datasetincludes a first subset of WSIs of tissue acquired from patientsdemonstrating clinically diagnosed heart failure, and a second, disjointsubset of WSIs of tissue acquired from patients that have not beenclinically diagnosed with heart failure; splitting the training datasetinto k-1 groups, where k is an integer; training the CNN with the firstk-1 groups; testing the CNN with the remaining group; and upondetermining that all patients have been used for training the CNN andtesting the CNN: ending the training.
 9. A non-transitorycomputer-readable storage device storing computer-executableinstructions that when executed by a computer controls the computer toperform a method for predicting cardiac failure, the method comprising:accessing a digitized whole slide image (WSI) of cardiovascularhistopathology; generating a pre-processed WSI by downsampling the WSI;extracting a set of non-overlapping regions of interest (ROIs) from thepre-processed WSI, where the set of non-overlapping ROIs has an oddcardinality; providing the set of non-overlapping ROIs to a deeplearning convolutional neural network (CNN); receiving, from the CNN, aprobability that a member of the set of non-overlapping ROIs afailure/abnormal pathology ROI; and controlling a computer assisteddiagnosis (CADx) system to classify the region of tissue represented inthe WSI as a non-failure histopathology or as a failure/abnormalpathology histopathology based, at least in part, on the probabilitiesassociated with members of the set of non-overlapping ROIs.
 10. Thenon-transitory computer-readable storage device of claim 9, where thedigitized WSI represents hematoxylin and eosin (H&E) stained tissuederived from a transplant, an explant, a surgical tissue sample, or anendomyocardial biopsy.
 11. The non-transitory computer-readable storagedevice of claim 9, where downsampling the digitized WSI includesdownsampling the WSI to an apparent magnification of 5×.
 12. Thenon-transitory computer-readable storage device of claim 9, where theset of non-overlapping ROIs is selected randomly from the WSI.
 13. Thenon-transitory computer-readable storage device of claim 12, where theset of non-overlapping ROIs includes 11 non-overlapping ROIs.
 14. Thenon-transitory computer-readable storage device of claim 9, where amember of the set of non-overlapping ROIs has dimensions of 256 pixelsby 256 pixels.
 15. The non-transitory computer-readable storage deviceof claim 9, where the CNN accepts one-hundred 64 pixel by 64 pixel inputpatches per member of the set of non-overlapping ROIs.
 16. Thenon-transitory computer-readable storage device of claim 15, where theCNN is a seven-layer CNN.
 17. The non-transitory computer-readablestorage device of claim 16, where the CNN comprises: a first layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 3,and a stride of 1; a second layer comprising a convolutional layer, abatch normalization layer, and an activation layer, the convolutionallayer having 16 kernels of size 2, and a stride of 2; a third layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 3,and a stride of 1; a fourth layer comprising a convolutional layer, abatch normalization layer, and an activation layer, the convolutionallayer having 16 kernels of size 3, and a stride of 2; a fifth layercomprising a convolutional layer, a batch normalization layer, and anactivation layer, the convolutional layer having 16 kernels of size 3,and a stride of 1; a sixth layer comprising a convolutional layer, abatch normalization layer, and an activation layer, the convolutionallayer having 16 kernels of size 4, and a stride of 2; and a fullyconnected layer having 2 kernels of size
 5. 18. The non-transitorycomputer-readable storage device of claim 10, where the CADx systemclassifies the region of tissue based on a majority vote of theprobabilities associated with members of the set of non-overlappingROIs.
 19. The non-transitory computer-readable storage device of claim10, the method further comprising training the CNN, where training theCNN includes: accessing a training dataset, where the training datasetincludes a first subset of WSIs of tissue acquired from patientsdemonstrating clinically diagnosed heart failure, and a second, disjointsubset of WSIs of tissue acquired from patients that have not beenclinically diagnosed with heart failure; splitting the training datasetinto k-1 groups, where k is an integer; training the CNN with the firstk-1 groups; testing the CNN with the remaining group; upon determiningthat all patients represented in the training dataset have been used fortraining the CNN and testing the CNN: ending the training.
 20. Acomputer-readable storage device storing computer-executableinstructions that, in response to execution, cause a computer assisteddiagnosis (CADx) system to perform operations comprising: accessing adigitized whole slide image (WSI) of cardiovascular histopathologyacquired at 20× magnification; generating a pre-processed digitized WSIby downsampling the digitized WSI to 5× apparent magnification;extracting a set of eleven non-overlapping regions of interest (ROIs)from the pre-processed digitized WSI, where a member of the set ofeleven non-overlapping ROIs has dimensions of 256 pixels by 256 pixels;providing the set of eleven non-overlapping ROIs to an unsupervised deeplearning convolutional neural network (CNN), where the CNN acceptsone-hundred 64 pixel by 64 pixel patches from a member of the set ofeleven non-overlapping ROIs, where the CNN is a seven-layer CNN havingless than 14000 neurons, where the CNN is trained using three-fold crossvalidation using a training dataset of WSIs of left ventricular tissueacquired from at least 200 patients, and where the at least 200 patientsinclude a first cohort diagnosed with end-stage heart failure, and asecond, different cohort without heart failure, where a WSI associatedwith the first cohort is labeled as failure/abnormal, and a WSIassociated with the second cohort is labeled as non-failure; receiving,from the CNN, an image-level probability that a member of the set ofeleven non-overlapping ROIs is a failure/abnormal pathology ROI; andcomputing a patient-level probability that the cardiovascularhistopathology represented in the digitized WSI is non-failurefailure/abnormal histopathology based, at least in part, on a majorityvote of the probabilities associated with members of the set ofnon-overlapping ROIs.